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Abstract 

The rigorous version of the result that unifies Maxwell-Boltzmann, Bose-Einstein 
statistics and Zipf-Mandelbrot Law is presented in the paper. Additionally, the 
Fluctuation theorem is included. Both results are set in the rigorous probabilistic 
framework. The system under consideration is composed of fixed number of parti¬ 
cles and energy smaller than some prescribed value. The particles are redistributed 
on fixed number of energy levels and are indistinguishable within one level. Fur¬ 
ther, the degenerations of energy levels are included and their number increases 
as the number of particles increase. The three distribution mentioned in the title 
corresponds to the three asymptotic cases of rate of increases of degenerations as 
a function of particles. When we take a thermodynamic limit for three cases we 
basically obtain the Law of Large Numbers and the obtained three means are our 
distributions. The fluctuations, i.e. the deviation from the mean turns out depends 
on the Entropy of the system. When its maximum is in inside of the domain we 
get Gaussian distribution. When it is on the boundary we get discrete distribution 
in the direction orthogonal to the boundary on which maximum is situated and 
Gaussian in other directions. The proof is similar for both results. The main part 
are the properties and asymptotic behavior of Entropy. Due to this we can optimize 
it and therefore apply probabilistic results which underlying assumptions overlap 
with ours and eventually get the desired results. 

Key words — Bose-Einstein, Maxwell-Boltzmann, Zipf-Mandelbrot, law of large 
numbers, fluctuations, Entropy 

1 Introduction 

Maxwell-Boltzmann and Bose-Einstein statistics are one of the most known results of 
Statistical Mechanics. The Zipf Law in turn, widely occurs in Complexity Science. For 
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this basic results we developed an interesting rigorous results which unifies them under 
one framework. Additionally we provide a corresponding fluctuation theorem and de¬ 
velop our results in the formal probabilistic manner. 

The underlying probabilistic system is defined independent of the context of the ap¬ 
plication. However, for the illustrative purposes we consider physical context, namely, 
the ideal gas. We consider the system of number of particles which has an average energy 
of particle smaller or equal to some fixed value. It is similar to the microcanonical en¬ 
semble, but the energy is not fixed, it can have some prescribed range of values. However 
our results can be easily adopted to the case when value of energy is within some small 
bounded interval. Further, there are number of fixed energies which particles can have, 
i.e. energy levels and the number of these energy levels is also fixed. Each energy level 
has a number of level degenerations. These degenerations has an influence when counting 
possible number redistribution of particles over energy levels. The Entropy of the system 
is a product of logarithms of combinatoric formula counting the possible configuration on 
each energy level assuming indistinguishability of particles having the same energy. 

There can be three types of system. We assume that the number of degenerations 
depends, increases, if the number of particles increase. We consider three cases depending 
on how strong this increase is. It can increase in the same way, slower or faster. This idea 


is introduced by Maslov 2 005a ], For each of theses cases we obtain different statistics, 


respectively Bose-Einstein, Maxwell-Boltzmann and Zipf-Mandelbrot Law. Normally the 
degenerations correspond to the fact that there are energy levels which differs by very 
small energy values and therefore they are considered as the same level. In our case 
there is an open question what physical interpretation can have this changing number of 
degenerations. 

The probability space which underlies the results is following. The sample space con¬ 
sists of the configuration of particles over the energy levels, divided by total number of 
particles. Theses configurations are constraint by fixed average energy and total number 
of particles. The sigma algebra is discrete and probability mass function of particular 
element in sample space is its entropy normalized by the partition function of the system. 

Our first result is weak law of large numbers. The distribution of particles on energy 
levels converges to mean when the number of particles tends to infinity, i.e. in thermody¬ 
namic limit. The three distribution mentioned in the title are obtained for different rate 
of change of the degenerations number. Additionally we provide a rate of convergence 
to t he mean. Simila r res ult but not mathematically rigorous was introduced in Maslov 
|2005a ] and Maslov |2005bl ]. The second result gives the distribution of the fluctuations 
from the mean as system tends to infinite size. Since the Entropy can have a maximum 
on the boundary or inside of the domain, depending on relation of fixed average energy 
of particle to average energy of enemy levels, we have two cases. The idea of two cases 
was first mentioned in Maslov 2005aj . For the maximum of the Entropy in the interior of 
the domain we have Gaussain fluctuations. When the maximum is on the boundary we 
have discrete distribution in direction orthogonal to the boundary on which maximum is 
situated and Gaussian in other directions. Here we also provide a rate of convergence to 
the limiting distribution. 

The proof for both results is similar. The essence of the proof are the properties of the 
Entropy and its asymptotic behaviour. Due to them we are able to optimize the Entropy 
and therefore distinguish the two types of its maximum and find the stati stics exp licitly. 


Similar calculations but partially rigorous and not complete were done in IMaslovl [2004 


xpa 


The asymptotic properties of Entropy and the considered probabilistic system overlap 
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with the assumptions of the probabilistic results in Kolokoltsov and Lapinski 2015]. We 


apply the theorem from that paper for both types of maximum and for three cases of 
degenerations and obtain our limit theorems. 

The paper is divided into several sections and appendix. The next section, second 
one is an introduction of the mathematical setting. We introduce there an underlying 
probabilistic system, a random variable and three cases of degenerations. In the third 
section we introduce and proof the main results of the paper, two limit theorems. Fourth 
section is devoted to the properties of Entropy, put as a Lemma with formal proof. The 
last section consists of the optimization of Entropy with related results needed in the 
optimizing. In the Appendix we provide some relatively basic results needed through out 
the paper. 


2 Mathematical setting 

For given integers G,N > 0, real number E > 0 and mapping 

e : {1,2,... G } —* M we introduce a probability space. The elementary events are uni¬ 
formly distributed G-dimensional vectors of nonnegative integers rq, i = 1,..., G satisfy¬ 
ing constraints: 


N = ni + n 2 + ... + n G , (1) 

EN > £{X)tL\ + £(2)tT2 + ... + e(G)tiq. (2) 

In physics we call such system micro-canonical ensemble. 

Arbitrary elementary event can be illustrated as the random distribution of N balls 
in G boxes. Moreover, each box has ’weight’ coefficient e(i) and the total ’weight’ must 
be less or equal EN. 

Furthermore, let us denote the image of the function e as the set {ey, £ 2 ,..., £ m } and 
without loss of generality it can be ordered E\ < £2 < ■ ■ ■ < £ m - To each element in the 
set corresponds a positive integer Gi, i — 1, 2,..., m representing the number of points 
in the domain of e having the values £*, so that G = Y17=i Gi- 

We can use this setting to define probability space in an alternative way. We consider 
the values Gi and £$, i — 1,..., m instead of the mapping e. Respectively, the conditions 
(HJ) and (J2J) are reformulated 

N — N± + N 2 + - - - + N m , (3) 

EN > £ 1 Ni + £ 2^2 + • • • + £ m N m , (4) 


where N t = n Gl+ ... +G ._ 1+1 +... +n Gl+ ___ +Gi _ 1+2 + n Gl+ ... + G i - 1 +G i for i = 1,..., m. This 
situation, can be illustrated as distributing N balls over m ’bigger’ boxes, where to each 
corresponds unique value £j. Then in each i-th ’bigger’ box balls are distributed over Gi 
boxes. 

For given vectors A f = (W,..., N m ) and Q = (Gi, ..., G m ) the number of different 
combinations which can occur in such redistribution, exactly the logarithm of that num¬ 
ber is denoted by S(Af) and called Entropy. 

We count those combinations using formula from Combinatorics for the possible num¬ 
ber of unordered arrangements of size r obtained by drawing from n objects, 


S(M) = In n 
1=1 


(Nj + Gj- 1)! 

Ni\(Gi-l)\ • 


(5) 
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Let us consider the discrete random vector denoted by = (W, X 2 ,..., X rn ) where 
Xi = Ni/N, i — 1,..., m and respectively sample space given by transformed conditions 
(J3]) and (J4]) is given by 


1 = Xi + x 2 + • • • + x m , 

E > e^xi + £ 2 x 2 + ... + e m x m , 


1 2 

Xi G ^ N'N’' 


N -1 
N ' 


and denoted by Q?v,e and respectively entropy function 

C( AT\ 1 TT _ 

^^)='»n feAr)!(G> _ 1)r 

The probability mass function (pmf) of random variable Xjy is given by 

1 


Pr(X N = x) = 


d S(x,N) 


Z(N,E) 

where Z(N, E) is a normalization constant specified by 


Z(N,E)=J2 


,S(x,N) 


( 6 ) 


(7) 

( 8 ) 


which is a total number of elementary events in the sample space Qe- Sometimes Z(N, E) 
is called partition function. 

We are interested in the behaviour of random vector X as N —> 00 . Since the domain 
depends on N the standard pointwise convergence of the function does not apply. 
However, we overcome that difficulty. For a fixed point x of the domain we choose a 
sequence of points x(N) when N —> 00 . These points, for each N, are the closest points 
from the domain to the fixed point x. Since, the number of points of the domain increases 
with N and they are evenly distributed x(N) —> x. This way we obtain convergence for 
every fixed point of the domain and therefore have convergence analogical to pointwise 
convergence. 

We consider a particular case when G = G(N ) is an increasing function of N. More¬ 
over, for each N the components G* are equally weighted and their number m remains 
constant. Which means that for all N, Gi = giG(N) for i — 1,..., m and some constants 
gi such that Y1T=x 9i = !• 

We distinguish three cases of function G(N), depending on its asymptotic behaviour 
in TV — y og 


1) 

2 ) 

3) 


G(N) 

N 

G(N ) 
N 

G(N) 

N 


00 , 

c i 

—)■ 0, 


(9) 


where c is some positi ve constant. The idea 
the paper of Maslov Maslov 2005 a . 


of three asymptotic cases is adopted from 
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3 Main results 


Theorem 1 (Weak Law of large numbers). Let X N be the m-dimensional discrete random 
vector on the sample space LIn,e with pmf specified by As N —» oo the random vector 
X N converges in distribution to the constant vector x* = (x\, x%, ■ ■ ■, x^). The exact 
values of the components of x* depend on the sample space parameter E. 

Let 9 E= m 5X 1 9i£i, then 


a) When E\ < E < ge the components of x* are 

!) < = '! 

2 ) < = 777tS-7. */ 


3) X" = 


gA £i~\-is — 2 

9i 

A £i + V 


if 


G(N) 
N 

G(N ) 
N 

G(N ) 
N 


—» OO, 

—^ c, 
—^ 0, 


for i — l,... ,m and the parameters A and v are the solution of the system of equations 

m m 

1 = J^x*, E = ^ 


iX ; 


2=1 


2=1 


6/ Wien E > ge the components of x* are 


x* = gu i = 


Further, we have following estimates for each case of G(N ) given by (GJ) 


7VV2-3e y ’ 

N 


1) M x „(t)=tf’•+()( 

MxAZ)=e^ +0 ( G{N)J , 

2 ) M x J^=e^' +o( 


when 


> 


AT 


jyi/2—3e G(7V) ’ 

1 


, iV 
men 


G(N) N V 2 - 36 ’ 


jyl/2—3e 


when 


Nl/2-3e 


» C(JV), 


Afx„«) = e« *' +0(c(iV)), 


3)Alv„K)=^+o( 5 ^i 73r3 

^(e = ^ + o(^a). 


w/ien C ( # ) » ^1/2-36 = 


when 


G(N) 

G(fV) 1 / 2_ 3e " AT ! 


> 


, G(iV) 

men ——— 


AT ^ G(AT)V2-3,’ 


where e G (0, min{l/2m, 1/6}) is a constant, N —* oo, Mx N (£,) is a moment gener¬ 
ating function of the random vector X N . In the second case ^jp- = c + 0(c(AT)), c(AT) 
is a positive, decreasing function and we define f(x) g(x) •<=>■ lim^oo f(x)/g(x) = oo. 
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Proof. Let us include the first constraint of domain Qn,e © by replacing the last compo¬ 
nent of x, x m = 1 — ^ ie function S(x, N). The we can define a new m — 1 random 

vector X' N = (Ad,..., X m _i), where its pmf is © with the substitution x m = Xi - 

The corresponding mgf of X' N can be obtained from mgf of X^ 


M X „(Z) = E[e (TXN ] = S[e fTJC » +f ” (1 - Ari —= e s “M x(f 

( 1 °) 

Since the function S(x, N ) with x m = 1 — Yl'iLi ^ ias a properties given by the Lemma 1 of 
the Entropy results Section, therefore we can apply Theorem 1 from Kolokoltsov and Lapinski 
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We do it for all the cases of the G(N ) and obtain our theorem for the random 
vector X' N except the exact vales of maximums. The estimates for the r.v. X' N are 
following 


1) Mx^i' 

M X ’ N (e 


£ \ = e (p-U) T x* + o( _I_^ 

+ u l jyl/2-3e J ’ 


£ ) = , q( N \ 

\G(N)J ’ 


when 

when 


1 _ N 
AT/2-36 > G(N) ’ 

N _ 1 

G(N) > iW 2 - 3e ’ 


2) M X ,„ ((' - U = + O (^r), 

Mi;(r - U = + 0(c(N)), 


when NTJtXTe » 
when c(iV) » 


3) M X , (5 ' - U = + O( G(N) \ /2 - 3t ). 

Mx j, «' - U = + O ^, 


when 


G(iV) 
G(7V) 1/2 - 3e " ^ 


> 


. G(N) 

when — 77 — ^ 


N G(N) 1 / 2 - 3^’ 


We obtain the estimate for our theorem by reversing the transformation (fTUj) on the 
estimate 

e U e (£'+Zm) T x* = e e T x*-U(. l-arj-. = e e T x*-Ux* m _ = e i T x* 


where x* is m dimensional vector which is maximum of S(x, N) in the limit as N —> oo. 
The exact values of the maximum we obtain from the Lemma 2 of the Entropy results 
Section. Hence we obtained our theorem. □ 


For the discrete random vector Xn on the sample space Qn,e with pmf specified by © 
we define a m -dimensional random vector Y/v- We consider two cases of Y/v for two types 
of maximum of the function S(x,N ) which occurs in the pmf. Theses cases are defined 
by the Lemma 2 of the Entropy results section. The type of maximum is distinguished 
by the value of the sample space parameter E. The random vector has forms 

a) Y n = y/h[N)(X N — x *), when E >~ge 

b) Y n — N(Vi — v*) + Vn(V — v*), whenei < E <ge (11) 

where x* and x* = Tv* = T(v*, v*) is the maximum of the functions S on Hat. The trans¬ 
formation x — Tv is a rotation of the coordinate system such that the axis v\ is orthogonal 
to the hyperplane of Vn,e on which maximum is attained. In the new corridanate system 
we have a transformed random vector X N — TV = T(Vi, V). Additionally, we assume 
that for the second type of maximum the hyperplane on which maximum is attained has 
a rational coefficient, i.e. values ei,..., e m are rational. 
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Theorem 2 (Fluctuations theorem). As N — * oo the defined above random vector Y/v 
for the maximum type a) converges in distribution for all three cases ofG(N) to am — 1 
dimensional random vector with normal distribution Af(Q, D 2 s(x*)~ 1 ) and m-th random 
component Y m is given by a combination of other random components Y m = — ^ i ■ 

For the maximum of type b) Y N converges, for first two cases of G(N), to a mixture of 
m — 2-dimensional normal distribution J\f (0, D 2 s(x*)~ 1 ) along variable V and discrete 
distribution with pmf ^ xp h 5 Oh , where i is an index of points in the coordinate v\ on 

2^t=i expjis (x )) 

the lattice T(Q N ) starting form the maximal point and m-th component ofY N is given 
throught other components Y m = — in the limit, where bi are some constants. 

The matrix of derivatives D 2 s(x*)~ 1 ism — lxm — 1 and D 2 s(x *)~ 1 is m — 2 x m — 2 
matrix along v^, i = 2,... , m and s’(x*) is derivative of s along v\. Additio7ially we have 
the restriction, that the estimate is valid for a subsequence of integers N which elements 
can be divided by some integer q, where | with v* = T _1 x*. 

Furthermore, we have estimates for type a) of maximum 


1 )M y „(Q 

My AC) 


M T D 2 s(x*) A | _|_ Q 


Nl/2-3e 


e 2 


+ r D 2 s(x*)- + 


(l + O 


N 3/2 \\ 

W)))’ 


when 

when 


1 _ N 
N l ~^ > G(N) ’ 

N _ 1 

G{N) > A+ _3e ’ 


2) M Yn (0 = e^ 2 ^*)- 1 * ^1 + 0^- 


7V 1 / 2 —3e 

MyAO = e^ TD2s{x ^( 1 + 0(y/G(N)c(N)) ), 


when fifYTe > C ( N )t 

1 

when c(N ) > 


G(Ny/2-3e 

G(N) 3/,2 \ 


3) MyAC) = ‘;(i + of 

MyAC) = ei (TD2 ^'>-^l+0^ N jj , 


and for the type b) 


V^°° c is'(x*)+£ii , „ „ 

1 ) MyA()= ^^j etr*+•>-'' 
= 


1 + 0 


when 


1 


> 


G(N) 
G(lV) 1_3e " AT ’ 

1 


, O(TV) 
men ——— +> 


TV V2-3e 


TV 3 / 2 \ 

1+01 cm) 


AT G(AT) 1 - 3e ’ 


, 1 iv 

when — „ S> 


A+~ 3e G(N) ’ 


, ^ 1 
men _ S> 


G(JV) iV 1 - 3 ' 1 


2 ) 


^»=i e f 1 , o ( _I_ 

e is'(x*) e ^ ^ V N l / 2 ~ 3e 

E oo 

i=i e M T D 2 s(x*)-H 

Yf°° ie is '(x*) 


Wken JfiFTe » C ( iV )’ 


l + 0(v / G(AT)c(AT)) ), whenc(N )> 


A+~ 3 ^’ 


as N oo, where e e (0,min{l/2m, 1/6}) is some arbitrary small constant and M Yn is 
a moment generating function of random vector Yv- 
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Proof. We start we the proof of the case a). We reduce the dimension of the underlying 
random vector of denoted by Wv- By using the constraint of the sample space © we 
reduce the dimension from mtom-1, i.e. apply the relation x m = l—x\ —X 2 — . —x m _i. 

Then the first m — 1 components of Y m will be unchanged. The m-th component wil be 
equal 


Y m = VN (x n 


m— 1 


m— 1 


m— 1 


X r 


= VN(l-^2 ~ X m) = X i ~ YI Xi ~ X *m) = ~ Y Yi 


2=1 


2=1 


2—1 


2—1 


where x* m = 1 - YJLi ■ 

Since the function S(x,N) with x m = 1 — YllLi x % in the pmf (|7[) has properties given 
by the Lemma 1 in the Section on the Entropy results therefore the random vector 
(Yi, Y 2 ,..., Y m _i) with considered pmf and the sample space hi?y f. for the fi rst tw o case s 


of G(N) fulfils the requirements of the Theorem 2 from Ko lok oltsov and Lapinski [2015 
We can apply that theorem and obtain results stated by this theorem. 

For the case b) we again reduce dimension of the underlying random vector X at. Since 
the Y/v can alternatively can be represented as 


Yjv = NTf 1 {x - x*) + VNT~\x - x*) 

where T~ 1 x = v and the Tf 1 is 1-st row and T is composed of rows 2, 3,..., m of the 
matrix T” 1 . If we set x m = 1 — x i then we can introduce altered transformation 

T' such that the random vector (Y 1? Y 2 ,..., Y m _i) = Y ^ can be represented 


Yff = N(v[ - v'*) + y/N(v' - v'*) = NT[-\x' - x'*) + '/NT'- 1 (x' - x'*) 


where x' = (aq,... ,x m -i) and prime generally denotes reduction in dimension. The we 
have for each Y* 


Yi = NT[-\x' -x'*), 

Yj = \fNT'r l ( x ' - x'*) 

where j — 2,..., m — 1. 

The component Y m we can represent as 

Y m = VNT~\x - x*) 

( m— 1 

2—1 

( m— 1 

i= 1 

= VNa T (x' — x'*) 

where a, is some constant vector and tj- are components of T” 1 . The resulting expression 
however can be represented as following linear combination 



Y 

1 rr, 


bi 

Vn 


m— 1 

Yi + ^ biYi, 

i =2 
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where b±, b 2 ,..., b r 


are some constants. In the limit, as N —» oo the first component tends 
to 0. Since the function S(x,N) with x m = 1 — X^=7' x * i n the P m f (ED) has properties 
given by the Lemma 1 in the Section on the Entropy Properties results therefore the 
random vector Y' N with considered pmf and the sample space TIn.e for the thre e cases 
of G(N) fulfils the requirements of the Theorem 2 from Ko lo kol t sov and La pin ski 
We can apply that theorem and obtain results stated by this theorem. 
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□ 


4 Entropy properties 


Lemma 1. The Entropy S(x, N ) as N — » oo for each of the case ofG(N) given by (11 ), 
and for all x E Qe has following properties 

d 2 d 2 

-—-^S(x,N)< 0, ——-— S(x, N) — 0, wheniy^j, for all N (12) 

(J i L/ ^ yj tE 2, tx «X/ j 

lim S(x,N ) < 0. (13) 

N^oodxj 

Additionally, for the first two cases of G(N) and third respectively, we have 

1), 2) DS(x, N ) =N[Ds(x) + a(x)e(N)], 1 = 1,2, (14) 

3) DS{x, N ) =G(N) [L>s 3 (a;) + a(x)e(iV)], 

as N —>■ oo, where D is differential operator, a(x) some twice differentaible function of 
x and 


1) Si (a) = ^T 


i= 1 


l i 

Xi In-h Xi 

Xi 


2) s 2 (x) =J2 

i=l 


(x, + g,c) ln(ii + QiC) - Xi In r, , 


(15) 

(16) 


3 ) s 3 (x) = Yi 


2—1 


g % In x t + g t 


(17) 


When Xi = 0 for some i-th component of x then corresponding summmad in the formula 
for si,l = 1, 2, 3 is equal to 0. The error term, e(N) respectively for each G(N) is defined 


1) e(N) 
e(N) 

2) e(N) 
e(N) 

3) e(N) 

e(N) 



0(c(N)), 

O' 1 


G(N) 


oi'Any 


N 


, 1 N 

when n ^ g(W y 
, 1 N 
when N <<C GW 

when — c(N), 


when 


when 


when — <C c(N ), 

G(N) 


G(N) 

1 

G(N j 


> 




N ' 
G(N ) 
N ' 


(18) 
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where in the second case = c + 0(c(N)), c(N ) is a positive, decreasing function and 
we define f(x) S> g(x) lim a ,_ ! . 00 f(x)/g{x) = oo. 

The properties of S(x, N ) remain the same if we consider it as m — 1 dimensional function 
with m-th component of x defined as x m = 1 — x i- 

Proof. Since T(iV) = (N — 1)! we can write 


(.XiN + 9i G{N) - 1)! _ T{ Xi N + g t G(N)) 
( Xi N)\( gi G(N) - 1)! T(xiN + l)T( gi G(N)) ’ 

for i — 1,..., m. Further, let us introduce 


QfiN) = - Xi N + 9i G(N) + + 9i G{N) - 0 In + ^G(iV)), 

^i(N) = -XiN - 1 + ^ Xi N + 0 In (:XiN + l), (19) 

Oi(N) = - gi G(N) + (giG(N) - 0 In 9i G(N). 

First order approximation of gamma function by the Theorem 1 in the Appendix A, has 
asymptotic expansion 


r(A) ~ V2ne~ x+ix+ ^ hlX 


1 + 


+ 


12A 288A 2 


+ ... 


, A —y oo. 


For Xi > 0 expressions XiN+ gi G(N), XiN+1, gi G(N) are positive and increasing functions 
of N, the following Gamma functions can be approximated using the above asymptotic 
expansion 


r{ Xi N + gi G{N)) ~ v^e® i{JV) 
T(xiN + 1 ) ~ V2^e edN) 1 + 
r ( gi G(N)) ~ V2^e^ {N) 11 + 


1 + 


+ 


12( Xi N + gi G(N)) 288(xiN + gi G (N ))‘ 2 

1 1 


+ 


, x 


+ 


12(xiN + l) 288(xiN + l) 2 

1 1 


+ ... 


, XiN + 1 —» oo 


+ 


12 gi G(N) 288 (g t G(N)) 2 


+ 


, gi G(N) —> oo, 


where QfiN), \Eh(iV), QfiN) are given by (??). 

From the definition of asymptotic expansion we obtain first order approximation, valid 
for Xi > 0 and corresponding large N 


r( Xi N + gi G(N)) - v / 27re' I, *( iV) 

T(xiN + 1) - v / 27re 3><(iV) < 

r ( gi G(N)) - < K\ e 


A Aj \j/ 


12 ( Xi N + 1) 
1 


12 ( Xi N + 9i G(N)) 
V2ire^ (N) 


yfce*'™ 


12 gi G(N) 




where K^, K^, Ki : Q are some positive constants. Note that we can increase these con¬ 
stants that above inequalities are also valid for Xi = 0 and all N. Hence we can represent 


l N + g l G(N) 
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the above inequalities as 


0 ( 1 ) 


r (^ + Sl Gm-^e^ = - (xiN + giGm 
r(n N + 1) - v / 2ire®‘ < ' T) = — jT v / 2ire*‘ (iv) , 

r(®G(JV)) - = r 2 °G ) { N) ' / ^ ee ‘ (m - 


y/2ne' ! ' i( ' N \ 


( 20 ) 

( 21 ) 

( 22 ) 


valid for A" —> oo. Next we combine approximations (1221) and (|2T|) using Lemma 6 from 
the Appendix A 


r(xiAT + l)r{ gi G(N)) - 2ne* i{N)+ei(N) = 


0 ( 1 ) 


27Te ^(N)+e^N )+ 


12( Xi N + l)12 gi G(N) 

, °( 1 ) OdT^W+egiv) , °( 1 ) 

+ l2(x t N + 1) V 2776 + 12 gMNV 776 

which holds as N —* oo. 

Now depending on the asymptotic behaviour of G(N) given by (??) this can be reex¬ 
pressed as 


1), 2) r( Xi N + l)T( gi G(N)) - 2vre^ (iV)+0i(7V) = — 


0 ( 1 ) 


N[12(xi+l/N)12 gi G(N) 

^-V^F e ^W +e< W +-^-y^W+e^iv) 


27re n> i (w)+e I (7V) + 


12(xi + 1/AT) 

3) T(xiN + l)T( gi G(N)) - 27re* i(JV)+0i(7V) = 


12 gi G(N)/N 
1 


0 ( 1 ) 


27re 


*i(N)+ei(N) 


G(N ) [l2( Xi N + 1)12# 

0(1) + °Wy/2^ e *m+em 


+ 


12( Xi N/G(N) + l/G{N))'-- ' 12<7 

and if we introduce constant 0 *©(x) it can be simply stated as 

1), 2) r(^7V + l)T( gi G{N)) - 2ne^ N)+e ' w = -^*©(x), 

3) r ( Xi N + l)T( 9i G(N)) - 2ne^ +e ^ = —J— a* e (x), 


(23) 


G(N) 


valid as A" —> oo, where 


1),2) 0 *©(a;) = 


0 ( 1 ) 


_O ^(JVoJ+OilJVo) , _ 0(1) _^/2^p1'i(^ 0 )+©i(A r o) i 

12(*i + l/iV 0 )12^G(iV 0 ) + 12(xj + 1/JV 0 ) 


, Q(i) — ,/2^p^d N o)+ei(N 0 ) 

+ 12^O(iV 0 )/iV 0 V 716 

3) <x*0 (x) = 


12(^0 + 1)12^ 

y/ 2 ^ e ^i (*>)+©<(iv 0 ) 

12ft 


2 7re «'i(A r o)+©i(A r o) + 


0 ( 1 ) 


l2( Xl N 0 /G(N) 0 + l/G(iV 0 )) 


1 /2^ e «'i(A r o)+0i(A r o) + 
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Now use Lemma 7 from Appendix A to combine the asymptotic equations (l20lb (j23j) and 
after some basic manipulations obtain 


1 ), 2 ) 


r(xiAT + gi G(N)) 


T( Xi N + l)T( gi G{N)) 2tt e *m+em 


< 


< 




2 7 re’ I 'i( Ar )+0i( Ar ) yl — ^cr^e(x) \N 


:<r#e(x) + 


12 ( Xi N + gi G(N)) 


3) 


r( Xi N + g t G(N )) 


rfoN + l)r(^G(7V)) 27 re*dA 0 +e i (A 0 


< 


< 


y/2ne 9i ^ 


27re *m+eiW \l- ^-ayeix) \G(N) 


-o-ye(x) + 


12 ( Xi N + gi G(N)) 


valid for N > N 0 and if we introduce constant cr ii $^e(x) we can simply represent above 
inequalities as 


1\ 2^ F(XjN 4- gi G(N)) _ 1 $i(N)-'Sri(N)-e i (N) _ ®i(N)-'Sr i (N)-e i (N) 

J T( Xi N + l)T( gi G(N)) ~ N y/2^ 

(24) 


3) 


T( Xi N + gi G(N)) 




T( Xi N + l)T( gi G(N)) 

valid as A" —y oo with 

1 


G(N) y/2^ 


1), 2) oy,M,©(a;) = 

3) crj $^e(a;) = 


1 - Jh a *&( x ) 


cry@(x) + 


12 (xi + g iG(N 0 )/N 0 ) J 


-a^ e (x) + 


1 — GpVo) 0 ^ 0 ^) \ G ( N ) ~ 12( x iNo/G(N 0 ) + gi ) J 


when Nq is some large constant. Now we use Lemma 6 from the Appendix A to get 
expression for the m factors for the first and second case of G(N) 


n 


rfeJV + g,G(N)) 


n 


^(AO-^dAO-ehAO _ 


r { Xi N + l)T( gi G(N)) 11 

1 , , 1 


m—1 


m—j m 


znn 

i=1 * j =1 q m-j,j k =1 l=m—j +1 


—a, It) 1 fWHI-WW-eilW 
N <r k , m (x)^e 


valid as A" —> oo and if we introduce constant cr^vj,© (x) we can simplify above asymptotic 
inequality for all cases of G(N) to 


m n 


T(x,N + ft G(JV)) 


n 


3 ) n 


q r(x,N + i)r( ft G(jv)) 1J XU 
T(x,N+ 9 iG(N)) 


1 e *.(»)-*,(K)-e.(K) = 1 (l) 


n 






i= 1 


(25) 


l = \ r ( Xi N + i)r( gi G(N)) ii V2FF 


PJ 1 „*i(N)-#i(N)-ei(N) _ _1 


G(N) 


ct^q{x 


n 


(AT-)—(JV)—©i ( 
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where 


^ m m— 1 m—j m 

1),2) awe(x) = + Y Y II II 


3) cr$^©(x) — 


0 i =l 

1 


N 

j= 1 Qrn-j,j k=l 0 Z=m—j + 1 

m—1 m—j 


G(7V ( 


o) 


— Y[<n,we{x) + J2 Y n 


— ^ 11 G(N 0 ) 

= 1 r m-j, 3 k=l V ' 


ft n o’jfc,^©^)- 


i=l j= 1 (fn-3,j k= 1 l=m— J+l 

Now, let us rearrange asymptotic equation (l25|i in terms of the function S(x,N ) 


1), 2) exp [S'(x, TV)] = exp 


2=1 

m 


ln( 2 I r )-”'' 2 + £ (J>,(JV) - 'I<,!.Vi - 0 ( (JV) ) + In (1 + 


in( 2 I r )-”' 2 + £ (j> f (iV) - *i(JV) - e f (JV)) + In (l + 


3 ) exp[S(x, TV)] = exp 

Since ln(l + x) — O(x) when x —> 0 we can simplify above equations into 

1), 2 ) S(x, N ) = ln(27r)- m / 2 + Y Ui{N) - ^(N) - Q t (N)) + -U 5 (x), (26) 

i=l ' ' 1 

3) S(x, N) = ln( 2 Ir )-”' 2 + g (ti(iV) - %(N) - 0,(JV)) + 
as N —* oo where <75 (x) = 0 (< 7 $^©(x)). 

Now, let us prove first result of the lemma. We start by explicitly evaluating functions 
<f>, T and 0 using (TT9]i for the first case of G(N) 


S(x, N) =ln(27r)- m/2 + ^ 


2=1 L 


x,(N) + g,G(N) - - ] ln(iiJV + g,G(N))~ 


(27) 


x,N - - ) lnU,.Y + 1) - gfi(N) - \ng,G(N) + 1 


+ JjVsW- 


valid for x, G [0,1] and large enough N. 

Then we calculate second derivative along x*, for some % within avaiable range 


dx 2 


S(x,N ) = 


2 N 2 


Xi N + gi G(N) 
2N 2 


- { Xi N + 9i G(N) - 1 / 2 )- 
N 2 


N 2 


+ (xjiV + 1/2) - 


x.JV+1 ' v "*‘' ' '(xW + 1) 2 , 
For Xi = 0 the second derivative becomes 


+ 


■xiN + 9i G(N)) 2 
1 d 2 a s {xi) 


N dx 2 


(28) 

(29) 


21 

dx 2 


S(x,N) =ln(27r)- m / 2 


N 2 


AT 2 


- - -N 2 
[ gi G(N) 2 ( 9i G(N)) 2 2 


+ 


1 g 2 ag(0) 

N dx 2 


which is negative for any case of G(N) and large N. Note, that <75 when x* = 0 becomes 
a constant. We prove for x* 7 ^ 0 be transforming the second derivative (128)1 


d 2 

dx 2 


S(x,N ) =ln( 2 vr )- m / 2 


iV 2 (^G(iV) - 1) 


N 2 


+ 


x,iV + 9i G(N))(xiN + 1) ( Xi N + l ) 2 
N 2 1,1 d 2 a s ( Xi ) 


[x t N + 9l G(N)) 2 , 


N dx 2 


< 0 , 
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for large enough N. Note the for the third case of G(N) we get the same result. The 
difference lies only in the change from N to G(N ) in the last expression which does not 
have influence for large N. 

Further, it is immidiet to see that in the limit as N —> oo, when ay = 0 the sign of second 
derivative is unchanged. For positive values of ay we transform the second derivative to 
the form for three cases of G(N) 


1),2) ^S(x,N) =N 


( 9i ~ 1 /G{N)) 


(a uN/G(N) + 9l )(xi + l/N) lV(ay + 1/Nf 


+ 


N(xi + g i G(N)/N)‘ 2 


+ 


1 d 2 a s ( 


Xj 


N dxj 


— oo, 


d 2 


3) q^S(x,N) =G(N) 


+ 


( 9i - 1 /G(N)) 


(Xi + gi G(N)/N)( Xi + l/N) G(N)( Xi + 1/Nf 

1 


G(N)( Xi + gi G(N)/N ) 2 

Hence we get the first results of the lemma. 


1 d 2 a s {xi ) 


To get the last results of the lemma we first further transform the approximation of 
Entropy 


S(x,N) =ln(2vr)- m/2 + ^ 


i —1 


(xi(N) + gi G(N)) \n( Xi N + 9i G(N)) - Xi N\n Xi N- 


- gi G{N) In 9i G(N) - - ln( Xi N + 9i G(N)) - Xi N In 1 + —- - (30) 


ay N 


— - In(ayiV + 1) + - In gi G(N) + 1 




for the first case of G(N). 

Now, for each case of G(N) we perform calculations separately. 
1) We further transform the Entropy and eventually obtain 


S(x, N) = ln(2vr)- m / 2 + N 
1 


i= 1 L 


x t In —+ \ Xi + g. 


Xj 


G(N) \ 
' N 


In 1 + 


XiN \ 

HiG(N)J 


ln(ayiV + 9i G(N)) - Xi In ( 1 + 

Ari G(N) 1 . . 

N In rW- + 


x,NJ ~ 2h HX,N + 11 + 2V S ' G{N) + V 


+ 


N N 

where during transformation we substituted the constraint of the domain Qe (??). 
Then we approximate appropriate logarithm using ln(l + x) = x + 0(x 2 ),x —* 0 to 
get isolate the function si(x) 


S(x, N) = ln(2vr)- m/2 + N 
1 


i= 1 L 


9 i 


0(1) XiN 


Xi In — + Xi + 2 

Xi g 2 G(N) 


+ Xi 111 1 + 


XiN 


g,G(N )) 


\n(xiN + gi G(N )) - Xi In ( 1 + 

Arl G(N) 1 . . 

N '" N + N as{x) ’ 


x,N I _ 2V ln(x,JV + 11 + SV gMN) + V 


+ 
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The first two terms inside the bracket are components of the function s(x) other tend 
to 0. Expressions outside the bracket depends on N. Then we calculate the derivative 
along Xi of the previously obtained Entropy and get 


S'(x, N ) =N 


In — + In I 1 + 


Xi 


x t N \ 1 1 

9iG(N)J ~ 2 XiN + gi G(N) 


-In 1 + 


+ 


XiN J 2 XiN + 2 


+ N a s( x ) 


Then, we perform some convenient rearrangements of expressions 


S'(x,N) =Ns[(x) + N 

1 


N 


G(N) 

G(N) [ N 

r m— 1 


In 1 + 


XiN 


g,G(Nj) 


+ 


+ N 


N 


E - 


i= 1 


2x i + g i G(N)/N 


— N In I 1 + 


+ 


XiN J 2 Xi + 2/N 


with 


s[(x) = In —. 

Xj 


where the logarithms in the expression are bounded by a constant, independed of N. 
This can be seen when we apply the expansion of logarithm log(l + x) = x + 0(x 2 ) 
when x —> 0 to those logarithms. 

We can write above formula in the following way 


S\x,N)-Ns' 1 (x) = K hi (x,N)-, 


S'(x, N) - Ns[(x) = Ii\i(x, N) 


or alternatively with constants cr(x) 


N 


g{n y 


t 1 N 
when — S> 


N G(N) ’ 


, 1 N 

when — <C 


N G(N) ’ 


S'(x, N)=N 
S'(x, N)=N 


s / 1 (x)+cr(x) 

s' 1 (x)+a(a;) 


N 


N 


G(N) 


t 1 N 
when — S> 


N G(N) ’ 

t 1 N 
when — <C 


N G(N) ’ 


valid for N > N 0 , where cr(x) are defined for two cases 

XiN 0 


a(x) =0 


<j(x) =0 


^o 2 

r g ( n q ) 

G(N 0 ) 

[ No 

G(AW In f 1 + 

N 0 

V 


In 1 + 


9iG{N 0 


1 

+ I t: 


2 Xi + giG(N 0 )/N 0 2x 


+ ^ + 0 


X 2 Nn 


XjNp 

9iG(N 0 


+ 


G(N 0 

N 2 


"vV 1 1 3 

jri \2xi + gi G(N 0 )/N 0 + 2xi + 


x 2 Nn 


This result can be asly extended to the vector of derivatives DS(x,N). 
Hence we get the first case of the second result of the lemma. 
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2) For this case we also further transform 


S(x, N ) = ln(27r) m/2 + N ^ ^ In ^ + 9i 


G{N) 


, G{N) , G(N) 

— Xi m xs — (h ——— m Qi ——— • 

1 ^ jv N 


x,N J 2N 


— In (xiN + giG(N)) - x t In 1 + —- - — In ( Xi N + 1) + — In giG(N) + — + 




In order to isolate the function s 2 (x) we use asymptotic inequality = c+0(c(lV)), 
where c(N) is a positive, decreasing function 

S(x, N ) = ln(27r) _m/2 + iV ^ (^x t + in ^ + gi cj + ^ + gi cj in ^1 + 


gic(N) in ( Xi + g, 


G(N) 




in (xiN + giG(N)) — x* in ( 1 + 


x,7V / 21V 


— ln(a:jiV + 1) + — ln^G(A^) + — + 




Then we calculate the derivative along Xi of the previously obtained Entropy and get 


m f / 

S'(x,N)=ln{2n)- m/2 + NY^ In ( 1 + 

i=1 L 


9iG(N) 

XjN 


+ In 1 + 


giG{N) 2 XiN + gi G(N) 


- hi ( 1 + 


XiN J 2XiN + 2 N 


+ — a six). 


Then, as for the hrst case, we perform some convenient rearrangements of expressions 


S'(x, N) = ln(2vr)- m / 2 + N c(N) 


Q(l)gi 1/1 1 

] Xi(l + f:C + x d ( N )) + N\ 2 x* + gi G(N)/N 


~ A ' ln l 1 + ^v) + 2^, + 2/Jv)] + h a ' s(x) ■ 

where the logarithms in the expression are bounded by a constant, independent of 
N. This can be seen when we apply the expansion of logarithm. Note that we have 
used J = c + 0(c(N)) in the transformation. We can write above formula in the 
following way 


S'(x,N)-Nsf 2 (x) = K 2>i (x,N)-, 
S'{x, N) - Ns' 2 {x) = K 2 ,i(x, N)c(N), 


or alternatively with constants cr(x) 
S'(x,N) = N s' 2 (x)+<j(x) 


S'(x,N)=N s' 2 (x)+<j(x)c(N) , 


when — 3> c(N), 
1 

when — <C c{N), 


when — S> c(N), 
when — <C c(N), 
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valid for N > Nq, with 


^ [ NoC{No) WTh) + L i + 9i G(N 0 )/N 0 + k +0 (^fk) ) ’ 

/ 0(l)gi 1/1 1 3 / 1 \\\ 

a{X) ~ 1^(1 + |c) + c(N 0 )N 0 \2 Xi + giG(N 0 ) /N 0 + 2^ + °\^N~ 0 )) J ’ 

as At —* oo and the second case is proved. 

3) First we transform (??) for the third case, i.e. the error has 1 /G(N) instead of 1/N 


m r / 

S{x, N ) = ln(2vr )- m/2 + G(N) £ ^ In a* + (, 


Xi TT7Tr\ + .9*1 ln fl + 1 - 9i In g t G(N) + 


G(N) 


1 N { 1 \ 1 

+ * hl N ~ 2G(N ) + " iG(iV)) “ ^ G(A0 hl i 1 + J " 2G(iV) + 1)+ 

^) ln9 ' G{N) + ^} + ^) as{x) - 

The function s 3 (x) can be isolated by applying logarithm expansion to the second term 
in the bracket. Then we differentiate along x t and perform appropriate rearrangements 

S '(,. N) = ln(27r) -m * 2 + G { N) £&©) + ( ~ 


i, °(i 2 Af) + 2x, + 2/n) + 


which can be represented as 


S'(x,N) = N S 3 (x)+cr(x) 


G{N)y 

g(n) : 

’ N ’ 


valid for N > N 0 , where cr(x) are defined 


S'(x,N) = At S 3 (x)+cr(x) 


, 1 G(N) 

when cm >> —- 

. 1 G(N) 

when cm <<: —’ 


w =0 ( J^_o f g - 2G(JY °b +1_1_+ _1 + o ( J-l 

1 J \G(N 0 ) V j 2 a;* + giG(N 0 )/N 0 2x, \x*nJ^ 

i i, nfnf g ? G(No) ) i ^ f 1 1 i 3 i nf 1 

a \ V xjNo J G(N 0 y\2 Xi + gi G(N 0 )/N 0 + 2xi \x?N 0 

Hence we get the error terms e(N) and last result is proved. 


5 Entropy optimization 

We consider maximization problem of the function S(x,N) over the domain Qg for all 
N and in the limit, as At —y oo. 
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Lemma 2. The function S(x,N ) for every N, when parameter of the domain E > £\, 
has a unique maximal point denoted by x*(N). As N —» oo the maximum exists only if 
the parameter of the domain, E > £\ and it is unique. 

For ge = A Yf r f =l gi£%, if E > ~ge, then the maximal point x* = (x\, x \,..., x* m ) has 
components 


x* = gt,i = 


and if E\ < E < ge, respectively for each case ofG(N), the maximal point has components 


1) < 

2 ) x* 

3) x* 


gi 

p\£i-\-V 5 

9i C 

gA^ 5 

9% 

A Si ~\~ v 


fori = 1,... , m, where the parameters A ? v are uniquely determined by the equations 


m 



1 , ^£iX*=E. 

2=1 


Proof. The domain is an intersection of two m-dimensional simplexes, first is de¬ 
termined by origin and standard basis vectors of M m , i.e., o,e i, . .. ,e m and second by 
vectors 0, E\e\/E ,..., E m e m jE. Since simplexes are convex sets, so their intersection and 
therefore the domain is convex. Since the function S(x,N) for all x E Ft e and N has 
properties f|l2|) . hence its matrix of second order derivatives D 2 S(x,N) is diagonal ma¬ 
trix with negative entries. Therefore it is a strictly concave function on a convex domain. 
When the domain is nonempty, E > E\, the unique maximum exists and we denote it by 
x*(N). 

In the limit N —> oo, in order to get maximum x* independent of N we consider 
function Hindoo jjS(x, N). Then the properties (fT3l) and (fl4l) of S implies 

,. 1 <9 2 S(x,iV) n 

Inn-—- < 0 

V—»oo N dxj 

for all x G FLe- From (fT 2 ]l we infer that in the limit the mixed derivatives are equal to 0. 
Hence the matrix of second derivatives \im.^^ f00 j^D 2 S{x,N) is a diagonal matrix with 
negative entries. Therefore is negative definite matrix and jfS(x, N ) is a strictly concave 
function in the limit. 

Since considered function is strictly concave, defined on a convex domain O^, con¬ 
straints (??) and (??) are affine, it is a convex optimization problem, for definition and 
terminology see Appendix A.4. Since objective is strictly concave the optimal vector, 
denoted as x* is unique, if exists. 

Now, we find the explicit form of the optimal vector. We start by representing our 
optimization problem in so called ’standard form’ 


minimize — //, (31) 

subject to e t x — E < 0, 
l T x - 1 = 0, 
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where 1 is the unit vector and £ = (£ 1 ,..., £ rn ). 

Since the considered problem is convex and if E > eq there exists x G Relint (id) , ( 
for the definition of Relint see Appendix A.4 ) with 

1 T x — 1 = 0 , 
e T x — E < 0, 

the Slater’s conditions holds (see Appendix on Theory of Optimization A.4) therefore 
strong duality occurs and optimal point exists. Then for each of the three cases of G(N) 
we have corresponding Karush-Kuhn-Tucker (KKT) conditions ( for details see Appendix 
A.4 ), 


e T x* — E < 0, 

1 T x* -1 = 0 , 

A > 0, (32) 

\(^£ T x* — E^j = 0 , 

— lim —DS(x*, N ) + X*£ + ul — o, 

N^oo N 

For the convex optimization problem with the strong duality, these are necessary and 
sufficient conditions for the vectors x* and (A, u) to be optimal. 

Now, we solve (l32]h Since 


lim —DS(x,N) 
N^oo N y ' 


Sl(x) 


with / = 1,2,3 respectively for each case of G(N). Hence for the first function si(x) = 
x, In + Xi from the last of KKT conditions we obtain 


X; 


9i 

g\Ei+v 


(33) 


Next, first four equations we represent as two cases for possible ranges of values of A. 
The first case is 


£ T x* — E < 0, 
l T x* -1 = 0 , 

A = 0. 

From Lemma 3 attached at the end of this section, the solution for the above system 
exists and is unique only if E > ~gE and (A, v) = (0, 0). Hence (l33ll becomes 

Xi = gi,i = 1, • • ■ ,m. 

The other case is 

£ T x* - E = 0, (34) 

l T x* -1 = 0 , 

A > 0, 
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where we have equality in the first condition because of A* ye T x* — EJ = 0. By Lemma 

3 of this section, solution (A, u) exists and is unique only if sq < E < ge. Further, 
substituting (1331) into two first conditions of (jM]) and we get the system of equations 
from which we can calculate parameters A and v explicitly. For the second and third 
case, situation is analogical but we use respectively Lemma 5 and 4. Only the outcome 
of the last KKT condition gives different result. □ 


5.1 Related results 

For the positive numbers g^e^i = 1 ,,m such that ei < e 2 < ■ ■ ■ < e m , X!”=i 9% = 1 
and some E > 0 we have the system of equations 

m 

= 1, (35) 

1=1 

m 

y £iXi = e. 

1=1 


where 


Xi > 0, i — 1 ,..., m, 


Let ge = ^ ^2'ILi 9i £ i then we following have lemmas 
Lemma 3. For the system of equations 1S5\) let 


= 9i 

gA Ei+V 1 


where A and v are some parameters. 
Then 


1 , • • • ,m. 


(36) 


(37) 


*) 

if E = gS, 

then the 

ii) 

if £i< E <ge, 

then for 

in) 

if~ge < E < e m , 

then for 

iv) 

if E £ (ei,e m ), 

then the 


solution is (A, u) = (0, 0) and is unique, 
A > 0 the solution exists and is unique, 
A < 0 the solution exists and is unique, 
solution does not exists. 


Proof. We start with proof of uniqueness of the solution (A, u). First let us assume the 
solution exists. Then we substitute (1371) into (1351) and get 




9i 




gA £i~\-U 7 


From the first equation we have 


v = 



(38) 

(39) 


(40) 
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Note that the function v = z/(A) is strictly decreasing, hence one-to-one. 
Next we substitute (140|) into second equation of (1381) and obtain 


E 


E m 

i=i9i£je Ae » 

Em \ ~. * 

i=i<7»e Ae - 


Let us show that E = E( A) is ono-to-one function. 
We calculate its derivative, 


E'( A) = 


-Er= 


1 ( Er=i 9^) + ( Er=i 




and represent as the difference of the expected values, 


E\ A) = E[£} 2 - E[S 2 }, 


where £ is a random variable with range = {eq,..., £ m } and pdf f(£i) = gi&~ X£i / E/li 9i e ~ Xej ■ 
By the Cauchy-Schwartz inequality E'( A) is strictly negative, therefore E = E( A) is 
strictly decreasing function, hence one-to-one. 

Assuming the solution exists, since E = E( A) and v = i/(A) are one-to-one, the solution 
(A, v) is unique. 

Now lets prove the existence of the solution (A, v). 

The function E( A) as A —* ±oo and A = 0 takes values 


E m —X f- 

i=l £ i9i e £l 

E m \ c . 

i=i 9ie 1 
as A —> oo, 



9i£i + T,?=2 £ i9i e Hei £l) 

9\ + E*U 9i^ x{£i ~ £l) 


E{ A) 
E(A) 


1 

m 


^ for A = 0, 

2—1 

gm£m + Ej'E 1 gj£ie~ x ( £i ~ £m) 

9m + Et~l l£ - Xi£i - £m) 


—^ <S r 


as A —> —oo. 


Since E = E( A) is strictly decreasing, points eq and £ m are boundaries of its range, hence 
A exists only if E G (£q,£ m ) and further 


if £\ < E < g£, 
if E = ge, 
if gE < E <£ m , 

Now, from (HU]) we have 

zz(A) —> oo 
v( A) = 0 
v( A) —> —oo 

hence for any A parameter v exists. 

Putting the results together we get the lemma. 


then A > 0, 
then A = 0, 
then A < 0, 


as A —> oo, 
for A = 0, 
as A —> —oo. 


□ 
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Lemma 4. For the system of equations $35 |) let 


i — •> . ) 

AEi + Z/ 


where A and v are some parameter. 

Then, 

i) 

E = ge, 

then the solution 

a) 

if Ei < E <gE, 

then the solution 

in) 

ifgE < E <E m , 

then the solution 

iv) 

if E ^ (si,s m ), 

then the solution 


i = 1, ..., m. (41) 


is (A, v) = (0,1) and is unique, 
exists and is unique for A > 0, A < —ue i, 
exists and is unique for A < 0, A > —i>E m , 
does not exists. 


Proof. We start with proof of uniqueness of the solution. First we assume it exists. Then 
we substitute (PXTh into (l35]h and get 


m 



1 = V 9l 

+ iF 

1=1 

(42) 


m 

E = V . £i . 

^ A Ei + U 

1=1 

(43) 

and then we perform a 

substitution v = Xa and for A ^ 0 we have 



m 

i = V 9i 

A (Ei + a) 

1=1 ' 

(44a) 


m 

« _ fh £ i 

. A(ej + a) 

1=1 v 7 

(44b) 

From (144ah we obtain 




A = V ® . 

“ £i + ol 

1=1 

(45) 

Except the singularities at the points a = —£i,i = 1,... ,m, the function A 
strictly decreasing. 

Note that by (136|) 

= A(a) is 


> 0, i - 

A [Ei + a) 

(46) 

hence A and a can take values 



A > 0, a > —Ei, 

(47) 


or 



A 0 ? cm 

(48) 


and if we define A = A (a) separately for the domains (1471) and 
Next we substitute substituting (j45|) into il 1 lbli and get 


it is also one-to-one. 


E 


E m 

i=1 


E m 

i=1 


9i£i 

£j+a 

9i 

Ei+a 
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Let us show that function E = E(a) is one-to-one. 
We calculate its derivative, 


E m 

i =1 


9i £ i 

( Ei+a ) 2 


E m Qj \ sprn g i£i / 

i= 1 £i-\ -ol ) Z-^i=l s-i -ha \ 


E'(a) = 

/ y^m 

Y £i+a 

and represent it in terms of the expectations 

E\a) = E[S}E 


9i 


(ei+a) 5 


1 

- E 

£ 

£ + a 

£ + OL 



where £ and (£ + ol) 1 are random variables with ranges 

{£+) • • • i £m\i 

1 1 


n 


(£+a)~ 


£\ + OL £ m + OL 


( 49 ) 


both with pdf f, = +++ ■ 

^3 = 1 £j+a 

Now, setting g(£) = £, h(£) = and use special case of FKG inequality, see Appendix 
on Probability A.3 for the details, 


E 


£ 

£ + a 


< E[£]E 


£ + a 


which implies E'(ol) is strictly positive, therefore E = E(a) is strictly increasing function. 
If we define E{a) for the domains (TX7T) and (FISjl separately, it is also one-to-one. Therefore 
the parameters A and a are unique. 

Next we prove the existence of A and a. 

Let us start with showing the existence of a. The function E(a) as a —* — £+, a —> —£ m 
and as a —> ±oo takes values 


E(a) 

E(a) 

E(a) 


1 gjSj 

a /-^i= 1 ii+1 

_ a 

1 gi 

OL 

E m gj£j(e i+a) , y^m gi£i(ei+a) 

i =1 Ej+a _ ~r l^j =2 £j +a 

E lia +a) , y^»n gj(ei+a) 

a=1 e^H-a 2-^i=2 

E m gi£i(em+a ) , y-'TO—1 giej(£m+a) 

i=i g^+a _ i~ Z^i=l gj+Q 

E lia (£m +a) n I gj (£m+a) 

*=1 £i+a ^ Z—/i=m—1 e^+a 


m 


^ ^ 9i £ ii 


i =1 


E m 

i =1 


9i£i 

£i+a 


E m 

i= 1 


9i 

£i+a 


as ct —> ±oo, 


as a —> —£i, 


as ol —y —£ m . 


(50) 

(51) 

(52) 


The equation (T5HT) implies that E(a) ^ A i g i£ ^ however if we take original system 
of equations, i.e. (H2]), then for (A, u) = (0,1) we have E = — Y^L\ 9i £ i■ Taking that into 
account and equations (l5Tf . (T52|) we get that a exists if E e (ei ,£ m ). 

Further, since function E = E(a) is strictly decreasing, 


if £i < E < g£, 
if E = g£, 
if ~gE < E < £ mi 


then a > —£\, 
then a —> ±oo, 
then a < —£ m . 
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Now from (1451) 


A(a) —> oo 
A(ck) —y 0 
A(a) —> —oo 


as a —> — E\ 
as a —* ±oo, 
us ex —y —£ m , 


hence if a exists the parameter A also exists. 

Since v = j, A and a exists and are unique, then the parameter v also exists and is 
unique. 

Putting the results together we get the outcome of the lemma. □ 


Lemma 5. For the system of equations I135\) let 


= _ m_ 



.., m. 


(53) 


where A and v are some parameters. 
Then 


i) if E = ge, 

ii ) if E\ < E < ~ge, 

in) ifge<E< Em, 

iv) if E £ {£i,£m), 


then the solution is (A, v) = (0,log2), 

then A > 0, A < —vs i, 

then A < 0, A > —vE m , 

then the solution does not exists. 


Proof. As in this case the parameters A and v in (153|) cannot be factorized w provide the 
proof without full rigor regarding existence and uniqueness of parameters. We start with 
proof of uniqueness of the solution. First we assume it exists. Then we substitute (155j) 
into (153]) . and get 




9% 


2—1 


Ei + V _ 2 ’ 


* = £ 


e* 


2 — 1 


gA ^ 


and then we perform a substitution u — Xa and for A 0 we have 


i = E 


9i 


2 — 1 
m 




2=1 


e \(£i+a) _ ^ ’ 

Qi^i 

gA(ei+a) — ^ 5 


(54) 

(55) 


Note that by (1551) 

9i 

gA (ej+a) — ^ 

hence A and a can take values 


1, —, m. 


A > 0, a > —Ei, (57) 

or 

A < 0, a < —E m , (58) 
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For A = 0 from the system of equation (l54j) we get that v = log 2 and E = ~ge. Then we 
get that the solution exists if E e {gim^im, gm^m) as from the second equation of (T54|) 
we have 




9i 


i=l 


Ei~\-v _ ^ 


< 


E 

2=1 




gA(g^+a:) — ^ ’ 




E 

2=1 


9i 


^\£i-\-U — ^ 


> 


E 

2=1 


9iEi 


gA(ei+a) — ^ 5 


i.e. the weighted sum cannot exceed its highest element or be smaller than lowest. 

(???) Since E = E( A, v) is a strictly decreasing function w.r.t variable A we have that 
for £i < E <~ge the corresponding parameters A and v are in the regime given by (I57|h 
For the values of E in e m > E > ~ge we have the other regime Q58j) . Putting together the 
outcomes we get the final result. □ 


A Analysis 

Theorem 3 (Taylor’s expansion). Suppose that f is a real function on the nontrivial 
convex closed set A e R m , n is a positive integer, f{n — 1) is continuous on A, f^(t) 
exists for every t e A. Then there exists a point xq between x* and x, such that 


™ 1 i m B k f(r*) 

f( x ) = /o*) + dx . dx .... dx , On - oo« 2 - o • • • o» fc - o+ 

k =1 ' 11,12,—J*fc=l 11 * 2 ' ' ' 


(59) 


E 


d m f(x e ) 


ni ■—' , dx it <h„,.. .dx, 

2l,22,...,2 n =l 


0*1 - OOi2 -<)■■■ 0<m - 


where xe can be represented, xq = x* + 6{x — x*), 0 < 6 < 1. 
The k — th in the Taylor’s theorem can be represented 


E 


00 ) 


*1,*2,•••,*& 


=1 dx h dxi 2 .. .dx lk 


On - X* h )... (.x ik - x*) = D k f 0)0 - x *) 


<8 )k 


(60) 


where <g) is tensor product, D : f — > V/ differentiation operator and D®D( g).. ,<g).D = 
and -D® fc /0)U=Zf> = D® k f(x e ) is an operater such that D® k f(x e ) : R mfc —» R. 
Hence, for n > 4 expansion (J59|) can also be represented as 

/O) —f(x*) + Df(x*) T (x — x*) + (x — x*) T D 2 f (x*)(x — £*)+ 

n-l 

+ y TlD® k f(x*)(x - x*)® k + -D® n f (xo)(x - x*)® n , 

' k\ n\ 


where 0 < 6 < 1. 

Since operator D® n f(xe ) is finite dimensional, hence it is bounded and we have 

\D m f 0e)0 - < \\D m f(x g )\\\(x - rr*)® n |- (61) 
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Were we define 


\D® n f(xg)\\ = sup 


\D® k f(x e )(x- x*)® k \ 


which is a norm of the operator. Further, by the definition of the tensor product we have 
that 

< x 0 x, x 0 x >=< x, x >< x, x > 

which implies = \x\ k . Putting that fact together with (Ell) we get that n-th term in 
Taylor’s Theorem is bounded by 


\D® n f(x e )(x-x*)® n \ < \\D® n f(x e )\\\x - x*\ n . 

If the operator D® k f(x) is invertible, then we have x = D® k f(x)~ 1 y and define 

II n2 a N -l„ \D® k j{x)S | 

II D f(x) || = sup-j-j- 

y&B \y\ 

where B is a range of the considered operator. Then we have inequality 


(62) 


D m f(x)-'y\ < | \D‘f(x) 


-1 


which we transform taking into account relation x = D® k f(x ) 1 y 

1 


< \D® k f(x e )x\, 


\\D 2 f(x 

setting x = (x — x*)® k and including |(x — x*)® k \ = \x — x*\ k we get 

I rQ _ rg * I ^ 


D® k f(x e )(x-xT k \, 


II D 2 f(x 

which is a lower bound for n-th term in the Taylor expansion. 
Analogically to Theorem 1 we can define foilwing Taylor’s expansion 


(63) 


,9/Or) _ df Or*) , ^ ^ 

2^ jM 2^ 


d k+1 f(x*) 


dxj 


dXn 


, k\ . . ^ dxjdx h dx i2 ... dx % 

k =1 2l,l2, — ,Zfc = l J K 


( X h - 0(^2 -<)■■■ - <) + 


(64) 


1 

n! 


E 


<9 m+1 /0^) 


dxjdxi, dx io ... dxi 


On - <)0r i2 - <) ... (:r im - <J, 


where xg is somewhere between x and x* . The vector composed of elements of above 
expansion for j — 1 ,..., m can be defined as 


1 1 

Df Or) = Df Or*) + D 2 f(x*)(x - x*) + V —- a;*)®* + —D®^f(x e )(x - x*)® n , 

k\ n! 

k=2 


where we define operator D®( k+V) ffx) : R mi —> M m for k > 1 as 


[D®( k +Vf( x )(x-x*)® k ]. 


22 <9 fc+1 /(x) 

. dxjdx^dx^ ... dx ik 


-<)(® 


^2 


~ x i 2 j 


[x. 




o> 
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for j — 1,..., m. 

Analogically to the bounds of the remainder of the previous Taylor’s expansion (1621) and 
(1631) we have for D®i k+1 ^f(x) 

| D^ k+1) f(x e ){x - x*) 0k \ < \\D^ k+1) f(xe)\\\x - x*\ k . (65) 

Moreover, if D®i k+1 if(x) is invertible we have 

Further, we dehne another Taylor’s expansion, analogical to one in Theorem 1 as 


d 2 f(x) d 2 f(x*) , ^ j_ 22 

2^ u 2-^ 


d k+2 f( 


X 


dxjdxi dxjdxi 


k\ 

k =1 


dx 3 dxidx n dx l2 ... dx lk 


-(x h - x* h )(x i2 - x * 2 )... (x ik - x* )+ 


( 66 ) 


E 


d m+2 f(xe) 


n\ . A , dxjdxidxi , dxi 0 ... dxi 


On - x* h )(x i2 -<)... (x im - 


where again, xq is somewhere between x and x*. The matrix composed of elements of 
above expansion for j = 1,... , m, l — 1,... ,m can be defined as 


n —1 


D 2 fix) = D 2 fix*) + V -D®( k+2) f(x*)(x - x*)® k + -D 0{ - n+2) f(x e )(x - x *)® n , 

kl n\ 


k =1 


where we dehne D®i k+2 i f(x) : M mk —> M™ 2 for k > 1 as 


[D®^f(x)(x-xT k ] jtl 


y d k+2 f(x) 

. ^ 1 dxjdxidx^dx i2 ... dx lk 

l2,—,lk=l J 1-2 * 


for j = 1,... ,m, l = 1,..., m. 

Then again we have that 

\D 0 ( n+ Vf(x e ){x-x*) m \ < \\D®( n+ Vf(x e ){x - x*) 0n \\\x - x*\ n , 
and if D®( n+22> f(xe){x — x*) 0n is invertible we have 

< \D®^f(x e )(x-x*)® n \, 


\x — X 


||£)®(n+ 2 ) f[x e ){x ~ X*) 0n | 


where n > 1. 


B Asymptotic theory and approximation 

Let / : A —* M be a continuous function and A = (a, oo) for some a. 
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Definition 1 (Big O). The function f is of order O of the function g : A —>■ M as x —> oo 
if there exists is a constant K > 0 and xk G A such that for all x > xk 


1/0*01 < K\g(x )|, 


and we write it symbolically 


f(x) = 0(g(x )), x ->■ oo. 

Definition 2 (Small o). The function f is of order o of the function g : A —>■ M as 
x — > oo *//or all K > 0 there exists xk G A such that for all x > xk 


and we write it symbolically 


1/0*01 < K\g(x )|, 


/0*0 = o(^(x)), x -> oo. 


Definition 3 (Asymptotic equivalence) . The functions f and g : A — > M are asymptoti¬ 
cally equivalent as x — >■ oo if for all K > 0 there exists xk G A sach that for all x > %, 
/(x) 7 ^ 0 and g(x) Y 0 and 

/ 0*0 


9(x) 


<K, 


and we write it symbolically 


f(x) ~ g(x), x y oo. 


Definition 4 (Asymptotic expansion) . The formal power series y)°E n a n x n is an asymp¬ 
totic power series expansion of f, as x —>■ oo if for all m G N 


/(x) = ^ a n x n + 0(x ( m+1 )), x —* oo, 


(67) 


72—0 


and we write it symbolically 


f 0*0 E a n x n , x —y oo. 


n=0 


If first few coefficients of power series are known then we write 

n / \ . Qj\ &2 . 

/(x) ~a 0 H-h — + ..., x ^ oo. 

X X" 

Furthermore (67) can equivalently be represented as 

772 

/(x) = J^a n x _n + <r(x), 

72—0 

cr(x) = 0(x _< - m+1 ^), x —» oo. 


( 68 ) 


Let the function T(A) = (A — 1)! where ! is a usual factorial and A G N. For A G R it 
is defined through its integral form 


T(A) = / f-'e-'dt. 






Theorem 4 (Gamma function approximation). The Gamma function T(A) can be ap¬ 
proximated 


r(A)~e-U A (^ 


1/2 


1 + 


+ 


12A 288A 2 



Proof. See ?, p.60. □ 

Lemma 6. Given inequalities 


Ai - Bi = Ci, i = 1,... ,m, 


(69) 

(70) 


where m € N following inequalities holds 

m m m m— 1 j m 

n^-n«.-n«+E e n n 

i =1 i= 1 i =1 j=l fc=i )=/+! 

Proof. We start by introducing equality 

m mm 

l[A i = (A 1 -B 1 )l[A i + B 1 l[A i , 

i=l i=2 i=2 

which we obtained by adding and deducting Bi Y\ff =2 A i to nULi A i ■ 

Then again, we add and deduce, but this time BfB 2 n;= 3 and (^i - B i) B 2YiZ3 A i 
and get 

m m m m m 

l[A i = (A 1 -B 1 )(A 2 -B 2 )l[A l + (A 1 -B 1 )B 2 l[A i + B 1 (A 2 -B 2 )l[A i + B 1 B 2 l[A i . 

i=l i =3 i =3 2=3 z=3 

We repeat that step until all Aids in the products are replaced by (Ai — Bi), which 
eventually leads to the equation 

m m m m— 1 j m 

ib- 4 * - B 2 +e e nn (if. - B h)Bg (7i) 

i= 1 i= 1 i= 1 j= 1 Qm—j,j k=l l=j +1 

where J a sum over possible arrangements of the elements of the set {1, 2,..., m} 
into two groups, where elements does not repeat and within the group the order does 
not matter. First group is of the size m — j and second j and their elements correspond, 
respectively, to the indecies ik, k — 1,..., m — j and b, l — m — j + 1 ..., m. 

Then we substitute (1M?D and obtain 


m m m m —1 j m 

n 1 n /; n f a e e n n 

i= 1 i= 1 i= 1 j=l ( yn-j,j k=l l=j +1 

which is our result. □ 

Lemma 7. Given equalities 

A i — B i = cTf, (72) 

A 2 B 2 — cr 2 , (73) 

where cr\,o 2 G R are some constants, following equality holds 

A i _ B i _ _ B l&2 ° 1 

A 2 B 2 B 2 (B 2 <7 2 ) B 2 + CT 2 
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(74) 


Proof. First we divide (172]) by (l73|) and get 

Ai _ Bi + <j\ 

A-2 B 2 — (7 2 

and then we transform LHS 

Bi + ai _ Bi a 1 

B 2 + ct 2 B 2 + 02 B 2 + o 2 ’ 

then add and deduce Bi/B 2 and perform some manipulations 

B\ ex 1 B\ B\ Bi <j 1 

- 1 -=- 1 - 1 -= 

B 2 + (72 B 2 + (7 2 B 2 B 2 + <7 2 B 2 B 2 + <72 

_ Bi BiB 2 _ Bi(B 2 + cr 2 ) <J\ 

B 2 B 2 (I? 2 + cr 2 ) B 2 (.B 2 + cr 2 ) B 2 + <J 2 

_ Bi _ Bp[2 (7 1 

B 2 B 2 (F> 2 + cr 2 ) B 2 + cr 2 ’ 

and then we put it back to (1741) . change the side of B 1 /B 2 and obtain final result 

Ai _ -Bi _ Bia 2 _ <7i 

A 2 B 2 5 2 (S 2 + cr 2 ) B 2 + cr 2 

□ 


C Theory of Optimization 

Definition 5 (Optimization problem in standard form). An optimization problem in 
standard form has the form 

minimize fo(x), 

subject to fi{x) < 0, i — 1,..., m, 
hi(x ) = 0, z = 1,... ,p, 

The vector x G M m is the optimization variable of the problem. The function fo(x) 
the objective function. The inequalities fi(x) < 0 are called inequality constraints and 
equalities hi(x) = 0 are equality constraints. 

The domain for which on which objective function and all constraints is defined T> we call 
a domain of the optimization problem. Any point x G V is feasible if it satisfies the all 
the constraints. 

Furthermore, the optimal value p* is defined as 

p* = mf{f 0 ( x )\fi(x) < 0,i = 1,.. .,m, hfx) = 0,i = 1,... ,p}. 

and x* is an optimal point (vector) , if x* is feasible and fo(x*) = p*. 

Definition 6 (Convex problem). An optimization problem is convex problem if it is of 
the form 

minimize fo(x). 

subject to fi(x) < 0, i — 1,..., m, 
hi(x ) = 0,i = 1,... ,p, 

where / 0 , / 1 ,..., f m are convex. Furthermore requirements must be met 
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• optimized function f 0 is convex, 

• inequality constraint functions must be convex, 

• equality constraints must be affine. 

Definition 7 (Affine hull). We define affine hull by set of all affine combinations of 
points in some set A C M m is called the affine hull of A and denoted by aff(A): 


aff(A) = {6>ixi + ... + 6 k x k | x u ..., x k £ A} 


Definition 8 (Relative interior). We define relative interior of the set C as 

relinfiC ) = {x £ C | 3 r >o B(x,r) D affA C C} 

Theorem 5 (weak Slater’s condition). The Slater’s condition hold if optimization prob¬ 
lem is convex and there exists x £ relint/D) with 

fi(x) <0, i = l,...,k, ffix) < 0, i = k + 1,..., m. 


where fi are inequality constraints and first k of them are affine and relintfD) is relative 
interior of the domain. Moreover if Slater’s conditions hold then optimal vector (A*, v*) 
exists and strong duality occurs. 


Proof. See Bovd and Vandenberghe (2004 ]. p.227. 


□ 


Theorem 6 (KKT conditions for convex problem). The Karush-Kuhn-Tucker (KKT) 
conditions are 


< 0 , 
hfix*) = 0, 
A* > 0, 
A iffix*) = 0, 

m m 

v/(x*) + A yffix*) + v;vhi(x*) = o. 


For any convex problem with differentiable objective and constraint functions, any points 
that satisfy KKT conditions are primal and dual optimal and strong duality holds. 

Furthermore, if Slater’s conditions holds then KKT are necessary and sufficient con¬ 
ditions for the optimality. 


Proof. See Boyd and Vandenberghe 2004], p.244. 


□ 
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